Validity in Quantitative Content Analysis

Over the past 15 years, educational technologists have been dabbling with a research technique known as quantitative content analysis (QCA). Although it is characterized as a systematic and objective procedure for describing communication, readers find insufficient evidence of either quality in published reports. In this paper, it is argued that QCA should be conceived of as a form of testing and measurement. If this argument is successful, it becomes possible to frame many of the problems associated with QCA studies under the well-articulated rubric of test validity. Two sets of procedures for developing the validity of a QCA coding protocol are provided, (a) one for developing a protocol that is theoretically valid and (b) one for establishing its validity empirically. The paper is concerned specifically with the use of QCA to study educational applications of computer-mediated communication. The primary role of networked computers in higher education has shifted from presenting structured, preprogrammed learning materials to facilitating communication. In turn, the role of educational technology researchers has expanded to include the role of communication researcher. In the late 1980s, studies began to appear that incorporated new perspectives, new methods, and new techniques. One of the most promising was quantitative content analysis (QCA). Berelson (1952) defined QCA as “a research technique for the systematic, objective, and quantitative description of the manifest content of communication” (p. 18). In this context, description is a process that includes segmenting communication content into units, assigning each unit to a category, and providing tallies for each category. Bullen (1998), for instance, studied participation and critical thinking in an online discussion by counting the number of times each student contributed to the discussion and by assigning each contribution to one of three categories of critical thinking. By 1999, enough of these types of studies had been conducted in the field of educational technology that we were able to prepare a literature review (Rourke, Anderson, Garrison, & Archer, 2001). We used published reports to illustrate many of the basic issues of QCA, including objectivity, reliability, units of analysis, types of content, research designs, and ethics. Our intent was to summarize the experiences of researchers like us who were struggling to apply this unfamiliar technique in a disciplined and efficient manner. We found the technique promising but chided researchers on the rigor of their reports, particularly on the lack of reliability data. Those who persisted with the technique found their own way to these conclusions, and today